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Abstract 

In this paper, the authors describe the benefits and pitfalls of mixed methods 
designs. They argue that mixed methods designs may be preferred when evaluating 
professional development programs for p-K-12 education given the new call for 
accountability in making data-driven decisions. They summarize and critique the studies 
in terms of limitations of the studies, the measures utilized in the mixed methods design 
(quantitative and qualitative measures), and the overall analysis of the impact of this body 
of literature. Implications for researchers who want to apply mixed methods designs and 
evaluators who want to influence professional development planning will be outlined. 
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Professional development is a critical component of schooling in the 21 st century. 
In fact, the current demand for teachers to change their teaching practice is mediated by 
apparently conflicting mandates. For example, NCLB requires that states provide “high 
quality” professional development for all teachers but does not specify what constitutes 
high-quality professional development. The assumption underlying professional 
development activities is that teachers learn as a result of participating effective 
professional development. Effective professional development leads to changes in 
teachers and consequently changes in student performance and achievement. Successful 
professional development, therefore, implies documentation of changes in teacher skills, 
attitudes, dispositions, and student performance in grades p-K-12. 

Despite what has been learned about how teachers learn, professional 
development appears to be inadequate given the limited research on effectiveness. In fact, 
although we know that professional development has the potential to impact student 
learning, we do not know what and how teachers leam from professional development or 
the impact of teacher change on student achievement (Borko, 2004). Moreover, the 
demands for school systems to be accountable for student achievement engendered by No 
Child Left Behind has resulted in a plethora of state-wide assessment systems whose data 
itself is called into question (Foote, 2007). From her vantage point in the role as director 
of research for the New York Performance Standards Consortium, Ms. Foote cogently 
questions the assumptions underlying state-wide mandated assessment procedures and 
poses an alternative way to judge the quality of a student’s education. However, in an 
essay published in the January 2007 issue of the Phi Delta Kappan, even the eloquent 
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Ms. Foote fails to mention the responsibilities inherent in being accountable to 
professional development program effectiveness. 

These provide good reasons to develop solid research agenda to document 
effectiveness of professional development in America’s schools. Towards that end, what 
are the characteristics of effective professional development? The American Educational 
Research Association (2005) recommended that effective professional development has 
focus (e.g., on subject matter and student learning; links to curriculum materials 
teachers use as well as state and national standards, assessment and accountability 
systems; is experienced and sustained over time; and involves a significant number of 
hours. The elements to study when determining effectiveness of a professional 
development program were also articulated (AERA, 2005), including teachers (the 
learners), the facilitator (the guide for the teachers), and the context in which professional 
development occurs. Borko (2004) suggests that professional development be studied at 
several levels such as a single program at single site, a single program at multiple sites 
with multiple facilitators, and multiple programs at multiple sites with multiple 
facilitators (with this level representing the gold standard of generalizability). Most states 
and school districts do not have such systematic evaluation systems in place, even though 
there is general agreement that a good evaluation plan involves a comprehensive 
examination of classroom practices, includes an assessment of the impact on teachers’ 
behavior, and extends the documentation of impact to effects on student learning. 
Moreover, the systematic evaluation plan will be ongoing, incorporate beginning the 
assessment process at the beginning of the professional development program and 
continuing until beyond the completion of the program. 
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In order to study all of these elements using recommended guidelines, a mixed 
method approach is desirable. The National Staff Development Council (NDSC) (2006) 
recommends using multi-modal approaches and multiple sources of information to 
evaluate professional development including an assessment of the impact on student 
learning 

Multiple data sources is the preferred method to make data-based 
decisions regarding what professional development should be offered and 
how it should be developed and delivered (NDSC, 2006, p. ) 

Rationale 

How can evaluators make more informed data-driven decisions regarding the 
impact of professional development programs in p-K-12 educational settings? We argue 
that one way to study how staff development improves the learning of students can be 
found in mixed methods research designs (Collins, Onwuegbuzie, & Sutton, 2006; 
Creswell, 2003; Creswell, Shope, Clark, & Green, 2006; Tashakkori & Teddlie, 1998; 
2003). 

Method 

The authors conducted a search of literature on mixed method studies in 
professional development. Database sources included ERIC, Psychoinfo, Dissertation 
Abstracts International, and footnote tracking. The search included the following key 
words: evaluation, mixed methods designs, staff development, professional development, 
inserx’ice, k-12 schools, impact on student learning. Other data bases for medical 
education and nursing and human resource development were also searched. The 
boundaries included journal publications and the years 1995-2007 (note: 1995 was 
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selected because it was the first year that national standards for staff development had 
been articulated). The articles that emerged were read and evaluated into categories, such 
as theoretical, empirical, or expert opinion. In this paper, the authors focus only on results 
of the empirical studies that have been published within the last 5 years (2002-2007) in 
order to provide the most comprehensive analysis of mixed methods designs in 
professional development. 

Results 

The results yielded fewer than 14 empirical studies wherein the researchers used 
quantitative and qualitative analyses to assess the impact of a p-K-12 professional 
development program in American schools. Table 1 provides a brief summary and 
critique of the 14 studies, including names and year of the publication, the publication 
venue, the discipline or content area of the professional development program, the type of 
mixed method design, measures utilized in collecting both quantitative and qualitative 
data, and limitations. 



Insert Table 1. Empirical Studies (2002-2007)About Here 



As can be seen by inspection of Table 1,11 researchers started with a 
quantitative approach followed by the qualitative methods. When considering the rich 
array of choices available to mixed methods researchers (see typology created by 
Tashakkori & Teddlie (1998; 2003), this was a surprising result and may be an indicator 
of the lack of sophistication and experience with mixed methods on the part of 
professional development evaluators. 
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Most researchers within these 14 studies collected and reported measures of 
changes in teachers’ behaviors. However, only four included documentation of changes 
in student achievement (e.g., Heneman & Milanowski, 2004; Van Hanegham, Pruet , & 
Bamberger, 2000; and Vasquez, 2003). This, too, is a surprising result given the fact that 
the NSDC Standards for Professional Development Evaluation which describes the “gold 
standard” for determining effectiveness of professional development programs had been 
disseminated since 1999. 

Discussion 

Given the limited number of examples of empirical studies where researchers 
used multiple methods to analyze the impact of professional development programs 
including measures of changes in teacher performance as well as p-k-12 pupil 
achievement, and given the limitations noted in every study, why do the authors continue 
to recommend that mixed methods be used? 

First, our readings of the power of mixed methods (e.g., Creswell, ; Tashakkori & 
Teddlie, 1998; 2003) remind us that several benefits accrue. For example, mixed methods 
approaches can address research questions other methods, when used alone, cannot. 
Mixed method studies can more clearly represent divergent views (see studies 
enumerated in Table 1 in which researchers tapped typically marginalized stakeholders 
such as Van Henegham et al., 2000 inclusion of children in k-5 grades. Mixed method 
designs can extend quantitative data and answer research questions that otherwise are left 
unanswered. 

Second, the authors recently completed two studies where they used mixed 
methods to track teachers’ changes in a) teaching behavior through survey and 
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observation, b) attitudes towards the inservice content through interviews, and c) 
generating an hypothesized model or testing the validity of a professional development 
model (Cramer & Nevin, in press; Giordano, 2004; in press). Table 2 depicts two 
different visual representations of the researchers’ sequence in implementing a model 
development, implementation, evaluation, and model evaluation mixed methods study 
(Giordano, 2004; in press) and Cramer & Nevin’ s (in press) sequence of generating 
research questions suitable to both quantitative and qualitative methodologies, then 
implementing first the quantitative phase, then the qualitative phase, and the final 
synthesis of findings derived from both phases. 



Insert Table 2. MIXED METHOD DESIGNS: Two Recent Studies 



As can be seen from an inspection of Table 2, Giordano (2005) first conducted an 
extensive literature review from which a theme analysis yielded the principles of 
effective professional development. This led to the articulation of the Giordano 
Professional Development Model. Then the researcher collected quantitative data that 
included measures of teachers’ change in concerns about their using the innovation and 
qualitative data from interviews. The qualitative phase of interviews pre-post-follow-up 
then followed. Theme analysis of the rich descriptive text of teachers’ perceptions 
resulted in an assessment, in the teachers’ voices, of the model which emphasized several 
features derived from theories of adult development and the value of collaborative 
authentic inservice training which occurs over extended periods of time. Cramer and 



Nevin (in press) generated research questions that reflected the need for both types of 
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methodologies. They collected quantitative data on measures of teacher attitudes and 
beliefs toward co-teaching and qualitative data from interviews that allowed their 
attitudes and beliefs about co-teaching to be validated with the survey outcomes. 
Furthermore, observations of the co-taught lessons anchored the beliefs and attitudes in 
actual practice. Outcomes were obtained for both studies that one method alone could not 
have generated. For example, in the Giordano study, the combination of control group 
(post inservice completion of surveys by experimental teachers, those who participated in 
the inservice to learn the innovation, and a matched sample of teachers with similar years 
of experience and grade level who did not participate in the professional development 
program) and mixed methods design combined to form a powerful case for validation of 
the professional development model itself. 

In summary, these experiences combined with our immersion in the literature on 
mixed methods have convinced us that it is worth the effort to design and implement 
mixed methods designs in spite of the fact that to do so requires a complex repertoire; 
that is, the evaluator must leam both quantitative and qualitative methods. It requires 
understanding both quantitative and qualitative data analysis and interpretation. It 
demands that research questions and contexts must reflect need for mixed methods. 
However, the major benefit is that outcomes yield richer and deeper understanding of the 
impact of professional development. 

Implications 

We believe that the results of the review of the literature on mixed methods in 
professional development provide a strong case for selection of mixed methods as n 
evaluation framework. In this section, the authors pose several directions for planners, 
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trainers, and evaluators who, we believe, must take into consideration multiple factors 
within their context; for example, their expertise, time available, and resources at hand, 
However, we believe that when planners, trainers and evaluators complete the Specify 
Clarify Diversify -> Celebrate , they will gain the benefits that mixed methods 
promises to those who want to use data-based decision making procedures for 
professional development activities. 



Insert Table 3 Specify -> Clarify -> Diversify^ Celebrate About Here 



Planners, trainers, and evaluators can specify questions that require a mixed 
method approach to resolve and they can specify the quantitative and the qualitative 
measures they will collect to address those questions. Planners, trainers, and evaluators 
can clarify a timeline which prioritizes the approach to data collection (e.g., will both 
quantitative and qualitative data be collected concurrently or sequentially? They can 
create multiple visual diagrams of the plan so as to communicate to various stakeholders. 
Planners, trainers, and evaluators can diversify data sources, data collection, data 
analysis, and data interpretation to impact. They can distribute the responsibilities to 
those who implement the professional development (trainers), those who receive the 
professional development (trainees — teachers), and those whose lives are affected (the p- 
K-12 children and youth, and their families). 

In conclusion, we are reminded of the complexities of our natural world. For 
example, the Double Helix reveals. We share one of the images of the double helix 
structure of the DNA molecule (Retrieved January 29, 2007 from 
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http : //academy . asd20 . or g/kadets/lundberg/ dna. html) to remind us that the complexities of 
the DNA molecule were revealed by scientific inquiry, model development, and 
empirical studies. When a complex image is uncovered in the natural world, we can 
hypothesize that double helixes may be helpful representations of the complexities of 
human social interactions such as those that occur in professional development. We look 
forward to the new discoveries that await us when complex research designs are applied 
to professional development activities. 
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Table 1: 14 Empirical Studies (1999-2007) 



Author 

(Date) 


Publication 

Venue 


Discipline 


MM Design 


Measures 


Limitations 


Cramer & 
Nevin (2007) 


Teacher 

Education 

Special 

Education 


Inservice Education 


QUAN-> 

QUAL 


Survey Ratings on Two Scales 

Follow up Interviews with 
Volunteers 

Classroom Observations 


No measure of k-12 
pupil changes 

No control group 


Giordano 

(2007) 


Computers in 
the Schools 


Inservice Education 
Technology 


QUAL-> 

QUAN-> 

QUAL 


Theme analysis of Lit 
Review -> Model Development 
QUAN->Hall & Hord CBAM 
Levels of Use & Stages of Concern 
Questionnaire 

QUAL-> Teacher Perceptions 
(verbatim text analysis of 
interviews) on Impact of the SD 
Model 


No measure of k-12 
pupil changes 


Heitman 

(2006) 


DAI, 

Fordham 


School District 
Math Standards 
Implementation 
Project (Policy 
Analysis) - Impact 
of NY State Math 
Standards 


QUAN-> 
QUAL 
analysis of 
single year 
vs multi 
year 

participation 


Survey instrument 

Focus Group Interviews 

*NY Math Assessment data (grades 
4 & 8) 


No control group 


Heneman & 
Milanowski 
(2004) 


Peabody 
Journal of 
Education 


Human Resource 
Management 
Practices & 
Teacher 
Performance 


QUAN-> 

QUAL 


Measures of HR practices in place in 
2 county’s school districts 

Measures of Teacher Competency 


No control group 
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Competency 
Aligned with 
Student 
Achievement 




Measures of Student Achievement 

Qualitative analysis of how much 
alignment was in place 




Lewandowksi 

(2003) 


DAI, Indiana 
University of 
Pennsylvania 


Educational 
Psychology (Self- 
Efficacy) 


QUAN-> 

QUAL 


Personal Self Efficacy Scores on 
Woolfolk & Hoy (1993) Teacher 
Efficacy Sccde 

Scores on Leithwood (197) Nature 
of School Leadership questionnaire 

Follow-up interviews with subset of 
teachers 


No measures of k-12 
pupil changes 

No control group 


Liston (2005) 


DAI, Argosy 


Inservice Staff 
Development, 
Inclusive Education 


QUAL-> 

QUAN 


Hall & Hord CB AM Levels of Use 
& Stages of Concern Q + Teacher 
Perceptions on Impact of Model 


No direct measure of 
pupil changes 
No control group 


Moore (2005) 


International 
Journal of 
Mathematical 
Education in 
Science and 
Technology 


Constructivist 
Student centered 
(collaborative 
learning) impact on 
Undergraduate 
Math Education & 
Achievement of 
Ethnic Minorities in 
an Engineers 
Program 


QUAN-> 

QUAL 


Achievement of target group with 
matched control group of ethnic 
minority calculus students (GPA, 
Grades in Course, 
Failure/withdrawal rates) 

Voices of the student participants 
elicited through comparing written 
course evaluations of target group 
with matched control group of 
ethnic minority calculus students 


No ‘treatment integrity’ 
measure of the extent to 
which faculty actually 
taught differently (the 
calculus course within a 
constructivist 
collaborative learning 
framework vs faculty 
who taught the control 
group calculus course) 


Muijis, 


Journal of 


Leadership and 


QUAN-> 


Survey 


No direct measures of 
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Harris, 
Lumby, 
Morrison, & 
Sood (2006) 


Further & 

Higher 

Education 


leadership 
development for 
‘highly effective’ 
inservice providers 


QUAL 


Focus group interviews 
Individual interviews 


leadership actions 

No measures of k-12 
pupil achievement 


North (2004) 


DAI, 

University of 
Alabama 


Effective Teaching 
(Community 
College Faculty) 


QUAN-> 

QUAL 


Survey using Roueche,Roueche, & 
Milliron (2003) Five Star Faculty 
Survey 

Online Chat Sessions and F2F Focus 
Group Interviews 


No direct measures of 
college student 
achievement of faculty 
with positive 
perceptions vs less 
positive perceptions 


Nimkoff 

(2005) 


DAI, UNC 
Chapel Hill 


K-12 Teacher 
Implementation of 
Program for 
Students with 
Learning 
Difficulties 


QUAN-> 

QUAL 


Hall & Hord CB AM Levels of Use 
(LoU) web-based survey 
Telephone Interview to assess 
impact longitudinally 


No measure of k-12 
pupil changes 


Schoech & 
Helton 
(2002) 


Qualitative 
Social Work 


Course delivery 
(distance ed/online 
combined with 
traditional campus 
based) 


QUAN-> 

QUAL 


Qualitative measures analyzed with 
NUD*IST 

Quantitative measures analyzed with 
SPSS 

Graphic analysis of ‘online 
chatroom’ text 


No control group 


Van 

Hanegham, 
Pruet, & 
Bamberger 
(2000) " 


Journal of 
Education for 
Students 
Placed at Risk 


Math standards 
reform (notably: (a 
cognitive 
constructivist 
perspective on math 
learning) 


QUAN-> 

QUAL 


TIMSS data set 

Stanford Achievement Test-9 

percentile scores 

Teacher Performance measures 

evolving from Project IMPACT 

Project generated interviews with K- 


No treatment integrity 
measures to verify that 
teachers were actually 
using the cognitive 
constructivist pedagogy 
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2 students (6 per class within each of 
x schools) or randomly chosen 
groups of students in grades 3-5. 




Vasquez 

(2003) 


Science 

Educator 


Science Education 
& Science 
achievement of 
students considered 
at risk for school 
failure 


QUAN-> 
QUAL Case 
Study 


Measures of St. Louis Public 
Schools students (81% African 
American; 82% low SES) on two 
achievement categories 
(achievement gap associated with 
race/ethnicity is being closed) for 
Grade 3 science 

Interviews with multiple 
stakeholders during each phase 
(planning and implementation) 

Progress Monitoring (periodic 
review and feedback) of Selection of 
Curriculum Scope and Sequence & 
Program Implementation 


No specific measures of 
changes in teacher 
behaviors in classroom 
instruction (content or 
process) 


Womack 

(2004) 


DAI, Auburn 


Staff Development 
Standards & Staff 
Development 
program to 
influence teacher 
performance so as 
to increase Student 
Achievement in 
Russell County 
Schools 


QUAL-> 

QUAN 


Qualitative measures of program 
implementation -> identified high vs 
low quality programs 

Teacher Performance (changes in 
instructional behaviors that increase 
academic performance in the 
teaching of writing) 

Student Achievement (writing) 


No control group 





Quantitative Qualitative 
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Table 2. Visual Representation of Giordano (2005) SEQUENTIAL QUAL-QUAN- 
QUAL Design 



Data 



Conduct Literature 
Review 




MODEL DEVELOPMENT PHASE 

Analysis |\ Results 



Identification of 
Salient Factors; Face 
V alidity provided by 
Panel of Experts 




Model Articulated 
& Ready to 
Implement 




IMPLEMENTATION & EVALUATION PHASE 




Interviews 




Content Analysis ^ 


Comments re Evaluation \ 
of Model & Level of Use \ 


VALIDATION 
Of the Model 








/ 


Showing Stages of / ^ 

Development i 





Table 2 (Continued): 
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Visual Representation of Cramer & Nevin (in press) SEQUENTIAL QUAN-QUAL 



Design 
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Table 3 

Recommended Steps for Professional Development Program Planners, Trainers, and 
Evaluators 



Specify -> 


Clarify -> 


Diversify 


Celebrate Multiple Outcomes 


Questions that 
require a mixed 
method approach 


Create a timeline 
and priority of the 
two approaches 
(Concurrent? 
Sequential?) 


Data sources 
Data collection 


Those who implement the 
professional development 
(trainers); 

Those who receive the 
professional development 


Quantitative and 
qualitative data 


Create a visual 
diagram of your 


Data analysis 


(trainees — teachers) ; 


to collect 


plan 


Data 

interpretation 


Those whose lives are 
affected (the children and 
youth and their families), the 
Community, Administration, 
Funding Agents, etc. 
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Figure 1 

Image of the Double Helix Structure of the DNA Molecule 




*Retrieved January 29, 2007 from http ://academy . asd20.org/kadets/lundberg/ dna.html 




